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Abstract — The one-bit deletion and duplication channel is 
investigated. An input to this channel consists of a block of 
£ > 1 bits which experiences a deletion with probability p, 
a duplication with probability q, and remains unchanged with 
probability 1 — p — q. For this channel a capacity expression is 
obtained in the asymptotic regime where p + q = o(l / log I). As 
a corollary, we obtain an asymptotic expression for the capacity 
of the so called "segmented" deletion and duplication channel 
where the input now consists of several blocks and each block 
independently experiences either a deletion, or a duplication, or 
remains unchanged. 

I. Introduction 

Given an integer £ > 1 and two constants p, q G [0, 1] 
such that p + q < 1, the segmented deletion and duplication 
channel treats independently each consecutive length I binary 
input block in one of the following ways: 

• one bit is deleted with probability p, 

• one bit is duplicated with probability q, 

• the block remains unchanged with probability 1 — p — q. 

Conditioned on a bit being deleted (duplicated) in a particular 
block, the deletion (duplication) occurs randomly and uni- 
formly over the block. Hence, the unconditional probability 
that any particular bit is deleted or duplicated is equal to p/l 
and q/£, respectively. 

When I = 1, the segmented deletion and duplication 
channel becomes the standard deletion and duplication channel 
where each input bit is independently deleted with probability 
p, duplicated with probability q, and is left unchanged with 
probability of 1 — q — 

An input to the channel consists of s > 1 consecutive blocks 
of length £. The corresponding output is thus a binary string 
of known length between n — s and n + s where 

di-r „ 

n = s ■ t . 

Rate R is said to be achievable if, for any e > and s 
large enough, there exist 2 nR codewords and a decoder whose 
average error probability over codewords is no larger than e. 
Capacity is the supremum of achievable rates and admits the 
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asymptotic expression 

C= lim -m&xI(X n ;Y(X n )) (1) 

s^oo n x n 

according to Dobrushin's capacity theorem [2 Theorem 1]. 

Segmented channels with synchronization errors were in- 
troduced by Liu and Mitzenmacher in [9| where, following 
an algorithmic approach, they proposed a zero-error coding 
scheme and thereby established a numerical lower bound on 
the capacity of the segmented deletion channel (i.e., for q = 0). 

A difficulty in obtaining a tight single-letter characterization 
of C stems from the fact that the receiver does not know the 
error pattern, i.e., which out of the s blocks experienced a 
deletion or a duplication (albeit it knows the overall num- 
ber of deletions and duplications). As a consequence, errors 
"propagate" across blocks. 

A useful technique to derive upper and lower bounds on 
C is to reveal the receiver the error pattern E s = {Ei}f =1 
where Ei = —1 if the i-th block experienced a deletion, 
Ei = 1 if the i-th block experienced a duplication, and Ei = 
otherwise J4], fl4l . When this side information is provided 
to the receiver, each block can be considered in complete 
isolation and we obtain the so-called "one-bit" deletion and 
duplication channel. The capacity Csi of the one-bit deletion 
and duplication channel is the capacity with respect to a single 
length I block. We hence have the obvious upper bound 

C < Csi , (2) 

where 

Csi = \™*I{X l -Y{X 1 )), (3) 

where X e denotes a random input block to the channel, and 
where Y(X e ) denotes the corresponding output. 

A lower bound to C in terms of Csi can be obtained by 
using the argument of 1T41 Section II. C]. First observe that 

I(X n ; Y(X n ),E s ) < I(X n ; Y(X n )) + H(E S ) . 

Using that H(E S ) = sHb(p,q) where Ht(p^q) denote^ the 
entropy function — plogp— qlogq— (1— p— q) log (1 — p — q) , 
it then follows that 

C SI -jH b (p,q)<C. (4) 

2 Logarifhms are taken to the base 2 throughout the paper. 



Note that an analytical expression for Csi remains to be found 
and a numerical evaluation, for instance, via the Arimoto- 
Blahut algorithm, is computationally heavy already for mod- 
erate values of £, say £ > 17. 

In this paper, we provide analytical upper and lower bounds 
on Csi which, via (O and yield upper and lower bounds 
on C. These bounds are tight in certain asymptotic regimes 
yielding the main capacity results. 

Throughout the paper, the following notational conventions 
are adopted. A binary length n vector is usually denoted by 
a bold script, e.g., x, and its length is denoted by |x|. If we 
want to emphasize the length of a vector, we alternatively write 
x n . For computational convenience, we sometimes refer to a 
particular sequence x using its runlength description r(x) = 
(xx,{ri(x)}) where r-j(x) denotes its zth runlength^ For 
instance, the runlength description of 0100110 is (0, 11221). 

We use y -< x whenever y is a subsequence of x, i.e., 
whenever y results from the deletions of |x| — |y| bits of x. 

The next section contains our main results and Section [III] 
is devoted to the proofs. 



II. Main Results 



Let 



TO. UC1 

^si — 



„ I(XHa);Y(X e (a))) 



(5) 



where X l (a) = X\,X^, ■ ■ . , Xi refers to the Markovian input 
given by 



Pr(Xi = 0) = Pr(Xx = 1) = ~ 
Pr(X t ^ AVi) = a, 2<i<l, 



for some fixed parameter a £ [0, 1]. 

An explicit expression for the lower bound (0 in terms of 
the parameters I, p, q, and a is given in the appendix. 

Further, define 



U 



6cI p ■ {£ - 1) + q ■ log (2 e+1 - 2) 



(l-p-g)log£^ 6{0)1} ,2-^^» 



(7) 



where H(y(x 1 )) is the runlength empirical entropy of x l 

H(r(x 1 )) = - ^ log . 

i>l 

Proposition 1. For any p,q,a G [0, 1] such that p + q < 1 
and any integer £ > 1, we have 



L a SI <C S i<U. 



In Fig. CD 



Au,c SI (£) = max 



U-C. 



si 



p,q:p+q<l C, 



(8) 



(9) 



SI 



'Notice that 17Ji r i( x ) = l x l- 



A is „ c „OT[%] 
A,j, Cs , (()[%} 



Fig. 1. Relative differences between Csi and its upper bound U (given by 
Ai/ c SI (£)) and between Csi and its lower bound max„ Lg^ (given by 




Fig. 2. Relative difference between max a L^j and L^f . 



(6) and 



A 



Lsi,Cs 



max 

p,q-p+q<l 



Csi — max„ Lgj 
Csi 



(10) 



represent the relative difference between Csi, which is ob- 
tained numerically by the Arimoto-Blahut algorithm, and the 
upper and lower bounds U and Lgj, respectively, the latter 
being numerically optimized over a S [0, 1]. 

As we can see, these bounds are fairly close for a wide 
range of p and q. For instance, their difference with respect 
to Csi is at most 5% for any p and q such that p + q < 0.6, 
as long as I > 2. Moreover, numerical evidence suggests that 
both A^c S7 (£) and Al si ,c S i(^) ten d to zero as ^ °°- 

In Fig! |2 



^L SI {q,£) = max 
pe[o,i-9] 



max a g [0il ] Lg 7 -L° s j 
max QG[0i i] L^j 



(ID 



represents the relative difference between L%j and the opti- 
mized lower bound expression max Q Lgj as a function of q, 
for different values of £. As we observe, when either I or 
q decreases, non-uniform inputs perform significantly better 
than uniform inputs. 

We now turn to the case where there is no side information 
at the receiver. For comparing our results with related work, 
we restrict ourselves to the purely deletion case, i.e., q = 0. 
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Fig. 3. Upper and lower bounds on the capacity of segmented deletion 
channel for I = 8. 
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Fig. 4. Upper and lower bounds on the capacity of segmented deletion 
channel for i = 2. 



For this channel, a lower bound to capacity is obviously 

L a ^L a SI -H b {p,q)/£ 

by © and ©. 

Figures |3] and |4]represent the upper and lower bounds on C 
given by Z7 and max Q L Q for £ = 8 and I = 2, respectively. 
The difference between these bounds is particularly significant 
for p w 1/2. Indeed, this is partly due to the fact that 
the difference between the two bounds is lower by the side 
information H},(p,0)/£ which is maximal for p — 1/2. Also 
note that U may be better or worse than the numerical upper 
bound given in lfl4l . For instance, for I = 8 (Fig. [3} we have 
that U is lower than the upper bound proposed in [14| for 
p 6 [0, 0.6] whereas the opposite holds forp 6 (0.6, 1]. Finally 
note that U appears to be a very good approximation for Csi', 
the difference gets negligible for p < 0.6 when £ = 8 and is 
negligible for any p < 1 when £ = 2. 

Asymptotics 

In the regime of large blocks and small synchronization 
errors we haveQ 

4 We say that f(l) = 0(g(£)) if there exists a positive real number k such 
that |/(^)| < k ■ g(e) when t ->• oo. 



Theorem 1. i. For p and q such that p + q < 1, we have 

L&=i-^]og/ + 2(x-i) + |(jr + i) 

+ (p + g)0(r 2 ); (12) 

ii. When (p + q) log £ — ► 0, we have 

U=l-P±«\og£+P e (K-l) + l(K + l) 

+ 0((p + q) 2 (\og£) 2 /£); (13) 

where K = Y™Li ^ 1.28850 

iii. W/;e« (p + q) log £ — » 0, we have 

C SI =1 - ^ log^ + | (K - 1) + |(K + 1) 

+ (p + g)0(r 2 ) + 0(^±^log 2 ^). (14) 

We note that for p = 1 (and hence q = 0), the 1 — ^f^- term 
in < TT~4T > corresponds to the zero-error capacity of the one-bit 
purely deletion channel ([12, Theorem 2.5]). 

Note that p and q do not play symmetric roles in the 
asymptotic capacity expression ( fT4l . An intuitive explanation 
for this is as follows. From the length of the output block the 
decoder knows whether the input to the channel experiences a 
deletion, a duplication, or remains unchanged. If a duplication 
occurs, then the decoder also knows the number of runs in the 
input since duplication cannot change the number of runs. By 
contrast, deletion errors can erase a run completely, thereby 
increasing decoding ambiguity. From Theorem Q] and (0), we 
readily obtain the following asymptotic expressions for the 
segmented deletion and duplication channel: 

Corollary 1. i. For any p and q such that p + q < 1, we 
have 

C=l-(p + q) l -^+0(£- 1 ); (15) 

ii. When q = and p = 0(^ _1 ) we have 

L - 5 =1 + (p/£) log {p/l) - K x ■ (p/£) + 0(£- 3 ) (16) 

where K x * log(2e) - ~ 1.15416377; 

iii. When p = and q = we have 

L°' 5 =1 + (q/£) \og(q/£) + K 2 ■ (q/£) + 0(r 3 ) (17) 

where K 2 = J2f=i W ~ lo g(§) - 0.84583623. 

Note that the first three terms on the right-hand side of 
( TToT i correspond to the first terms in the asymptotic expansion 
of the capacity of the i.i.d. deletion channel with deletion 
probability p/£. 

III. Proofs 

We denote by pd and pi the unconditional probabilities of 
deletion and duplication, respectively, of each bit within a 
block of length i, i.e., 

Pd=p/£ Pi = q/£. 
Also, we denote by rt r (x) the number of runs in a sequence x. 

5 This constant appeared as Ai in (JJ Theorem 1], 



A. Proof of Proposition [7] 

1) Lower bound: The left-hand side of ((HJ holds because 
of©. 

2) Upper bound: For any length i output sequence y, we 
have P Y (y e ) = (l-p-q)P x (y e ) and Q(yV) = (1-p-q). 
For a length I — 1 (respectively, ^ + 1) output sequence y, 
resulting from a one-bit deletion (respectively, duplication) in 
the z-th run of x e , we have Q{y\x e ) — ^j 1 (respectively, SL j ± )- 
Thus, we can write 

I(X e ; Y(X e )) = P(YpC £ )) - #(Y(^)|X £ ) 
= (l-p-«)H(^) 

+ + J] P x (x) £ ^.log r ' ; 



x£{0,l}' ie{l,...,n r (x)} 

£ Py(y)logPy(y)- ]T 
|y|=*-i y:|y|="r(y) 



plogp + glogg, 



Py(y)logPy(y) 



(18) 



The sum of the first two terms on the right-hand side of the 
second equality is a concave function of Py. By the Lagrange 
multipliers method one deduces that the maximum is attained 
for the distribution 



E 



xe{o,i} 



2 P— 1 



Maximizing separately the third and the fourth terms on the 
right-hand side of the second equality in < fT~8b under the con- 
straints E|y|=*_i Pr(y) = V and E y: | y | = „ r(y) Py(y) = q 
is similar to entropy maximization and the maximums are 
achieved by the distributions 



Pv*{t- i ) = ^- l and P?*{y 



2 £+l 



respectively. 

Substituting distributions P£, P y *, and P y ** on the right- 
hand side of the second equality in ( fTSI l we obtain U. 

B. Proof of Theorem Q] 

i. This part of the theorem is obtained by deriving the 
asymptotic behavior of d27| i as I — > oo. To do this, we 
need the following lemma: 

Lemma 1. For any positive s,t such that s + t = 1, we 
have: 



fc=l 



0{- 



s-l 



(19) 



Proof of Lemma Q} This lemma is proved via 
the moment generating function method of (5). For any 
sequence of real numbers {fk}, the Bernoulli transform 
of fk is defined as 



Further, for fk and its Bernoulli transform S n , the gen- 
erating functions are defined by 

/(*) = x>«* and sw = E^ z "' 

fc>l n>l 

respectively. 

It is easy to check (see [5|) that / and S satisfy 



S(z) 



—fi-^-)- 
l-tz y l-tz ! 



Now we consider two sequences of real numbers fjp = 
logfc and f (2) = fclog/c, fc > 1. For and i e {1,2}, 
we denote the Bernoulli transform, generating function, 
and generating function of the Bernoulli transform by 
S ( n\ f {l) (z), and S®(z), respectively. Also, we denote 
by g' the first derivative of a function g. 
It is easy to check that f( 2 '(z) = z ■ (f^)'(z) which 
implies that 

S {2 \z) = -tzS {1) {z) + (1 - tz)z{S {1) )'{z) . 

Now, from [5 Propostion 1], we know that 

2sn n z 

Denote by the n-th coefficient of a gener- 

ating function A(z). Since [z n ]z k S(z) = S n ~k and 
[z n ]S'(z) = (n+ l)S n+ i, we obtain 

= _ t pog ( s(n - 1)) + " + 0{\)] 

+ n[\og(sn) + ^-— + 0( — )] 
Zsn n z 

s - 1 1 

= snlog(sn) + tloge H hO(-). 

2 n 

Since Sn ^ corresponds to the left-hand side of fl% the 
proof is complete. ■ 

For any p and q, as I — > oo, we have 



p + q^i-j + 3 

3=1 

{p + q)K 



( P + q)0(£- 2 ), (20) 



where K is defined as 



^ \k 

k=0 



fkS't 



k±n — k 



K= lim V 2-W +1 )jlogj. 



Also, we have 



(2 . 2 «-+ ^ vm 

m— 1 



= V^ (o - 5r(o - 5r " i Q) miogm 



= -'l + l\ogi + q o(e- 2 ) 1 



(21) 



where a follows from Lemma Q] by setting s = t = 0.5. 
By substituting (|20| and (EB into (|27]) we obtain ( fT2b . 
ii. Since the runlengths of a length i sequence are between 
1 and £, we have H(r(x 1 )) < \og£. If we assume that 
(p + q) log£ -4 0, we can use Taylor's expansion of 2~ x 
around x = to get 

(p + q) 



(l-p-g)loge 
+ 0((p + q) 2 £ 2 ). (22) 

Thus, we have 

j2 2-^^^)) = 2 i {1 _ 0{{p + q)2f)) 



(p + q) 



(l-p-q)\oge*- 



(23) 



Now, we establish the asymptotic behavior of 
Tl,x l H(r(x 1 )). Denoting by n(£,j), the number of 
times a run with length of j appears in all length £ 
sequences, we have 

f. 



J2H(r(x e )) = -J2n(£,j)jlog( J -) 



3 = 1 
£-1 

= -E 2 

i=i 



3=1 

-logW-E2 



J 1 J lo §7 



— 7 J log j 



£_j+3. 
7, J 



^2 e (log£-K), (24) 

where a follows from IfTT] Proposition 2] and where b 
follows from YljLi — 1- Therefore, we have 

log(E2-^ #(r(x<)) ) 



p + q 



l + \og(l-- 

(l-p-g)log2 

fO(((p+?) log^) 2 )) 



(logf-lf) 



t- 



p + q 
l-p-q 



(\og£-K) + 0(((p + q)log£) 2 ). 



(25) 



By substituting dZSl ) in @, we obtain ( TOi 
iii. The capacity expansion in (fT~4l > follows from ( fT2l . (1131 1. 



C. Proof of Corollary Q] 



i. Since 



H b {p,q) _ nif-i 



= 0{£ ), we have 



L^ = l- ip + f 0ge +Q(£-i). 



Also ([8]l , ( fT3l ) imply that C is upper bounded by 



1 



(p+q) logl nff-i 



0(£ ). The proof is complete. 



ii. We expand (1 — p) log (1 — p) around p = to obtain 

i°- 5 =i-?^-? + !iog(^) + ( > 



(i-p) 



£ °H ' y £' 
{-p + 0(p) 2 ) log e + pO(£- 2 ) 



= l+p d \og Pd - (log 2e - + 0(r 3 ) . 
iii. The proof is similar to the previous case. 
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Appendix 



For any p,q,a € [0, 1] such that p + q < 1, and any integer 
£ > 1 we have 



ra _ 



1 + (1 -p){£ - l)H b (a) + (p + q)(l - a) 1 - 1 \og£ 



2a 6 



2)-{£ 2 +£- 6)a 2 



£ 2 (1 -a) 

+ (£ 2 - U - 2)q + 2£j logo 

+ ( - 2a 3 (£ - 2) + a 2 (£ 2 + £ - 6) 

- 2a(£ 2 - 21 - 1) + - 3)) log(l - a) 

pa 2 (l-ay^ ^ \{t-A fo 



f 2 



( )™log(/? + 7 + 7™) 

1 — a 



E (1)(t^:)"'"'" 1( ^'" 



£ 2 (1 — a) ^— ' \ ?Tz 7 x 1 — 

v ' m— 1 v 

£ 2 

m=2 



k=l 



-jfe-1 
TO - 2 



fclogfc 



(26) 



where 



« 1 - 2a 
7 = - — and 



« £-l + (a 2 -a){2£-A) 



or a* 
When a = 1/2 the above expression reduces to 



m— 1 



p + g log! 
P + gy (^-j+3) 



x jlogj . 



(27) 



Proof: In order to prove ( I26l i. we need the following 
lemmas. 

Lemma 2. For any integer n > 1, we have 

( n \ k 2 -t k = n(l + t) n " x * + n(n - 1)(1 + t) n - 2 t 2 . 



(28) 

Proof of Lemma |2} The first and second equations can 
be obtained by taking the first and second derivatives with 
respect to t of the Binomial equation 

E^ = (!+*)"• 



n'(£,m) = 2 



(29) 



Lemma 3. • 77;e number of length £ sequences containing 
to runs is 

£-1 

V TO — 1, 

77je number of length k runs among all length £ se- 
quences containing to runs is 

(2 ifm = l,k=£ 

n"(k, m,£) = < 2m(^^ 1 ) ifm>2,k<£-m + l 
[ otherwise 

(30) 

Proof of Lemma \3} 

The number of length £ sequences containing to runs is 
twice the number of positive integer solutions of equation 



which is flU 



ri H h r m = £ 



£-1 

TO — 1 



(31) 



Since the only two sequences containing 1 run are the all- 
zero and all-one sequences we have n"(£, 1,£) = 2. The 
number of runs of length k among all length £ sequences 
containing to runs is twice the number of times k appears 
in the solution set of OTb . The number of times that the 
first run has length k is twice the number of positive 

integer solutions of r% -\ 1- r m = £ — k. Therefore, the 

number of times a run of length k appears in all length 
£ sequences containing to runs is equal to 2to(^'i~ 2 1 ). 



We write Lgj as 



a H(Y)-H(Y\X e (a)) 



Lsi t 

First, we calculate H(Y(X (a))). To compute this entropy, 
we need to calculate the probabilities of all output sequences. 
We classify the output sequences according to their lengths. 
For length £ sequences, we have 

Py{y") = {l-p-q)Px{y l ), 
which results in 

- £ P Y (y e ) log P Y (y e ) = (1 - p - q)H(X e (a)) 



(1-p-q) log(l - p-q) 
(32) 



where the input block entropy is given by 

H(X e (a)) = ! + (£- l)H b (a) . 



(33) 



Now, we turn to output sequences of length £ — 1. For any 
a € [0,1] and integers £ > 1 and to < £, we define 



f(t,m,a) =0.5(1 -a) 



l—m — l^rn 



(34) 



The probability of any sequence generated by a first-order 
Markov process is a function of the number of its transitions^ 
Since the number of transitions of a sequence x is equal to 
n r (x) — 1, for any length £ sequence generated by ©, we can 
write 



P x (x) = /(£,n r (x)-l,a) 



(35) 



To calculate we need to calculate the probability 

of each of its length £ super-sequencesQ A length £ super- 
sequence of y can be generated by inserting one bit into 1 
in one of the following ways: 

> Insert one zero (one) to one of its runs of zeros (ones). 
The number of distinct super-sequences generated under 
this scenario is equal to n r (y). Let x' be sequence y with 
one bit inserted in its i-th run. Hence we have Q(y|x') = 
( r i(y) + 1) ' Pd- Also, note that for any such x' we have 
n r (y) = rv(x') and thus, Fx(x') = f(£,n r (y) - l,a). 

• Insert one opposite bit at one of its ends. The number 
of possible super-sequences generated under this sce- 
nario is 2. For any such super-sequences x" we have 
Q(y|x") = pd- Also, note that n r (x") = n r (y) + 1 and 
thusP x (x") = /(^n r (y),a). 

> Insert one opposite bit inside of one of its runs. Since 
for any sequence of length £ — 1, there are £ + 1 super- 
sequences of length £, the number of possible super- 
sequences generated under this scenario is £ — n r (y) — 1. 
For any such x'" we have Q(y|x'") = pd and Px(x"') = 
f(£,n r (y)+l,a). 

Therefore, for any y £ {0, 1} , we have 

Py(j) =Y,Px{x l )Q{y\x l ) 

= (j8 + 7 • «r(y))/(*, «r(y) + 1, a)pd ■ (36) 
Hence, we have 

- J2 ^V(y)iogiV(y) 

ye{o,i}'- 1 

= -(Pilogp i )53(/3+7-n r (y))/(^,n P (y) + l,a) 
y 

: ip + t ■ My))f(^ My) + 1, «) x 



Pd 



log 



(i8 + 7'»r(y))/(Wy) + i.a) 



where 



-(Pd logp d ) • Ax -p d ■ (A 2 + A 3 ) 



= £(/3 + 7 -^(y))/(Wy) + i,a) 



(37) 



(38) 



6 Number of transitions of a sequence is the number of times its two 
consecutive bits differ 

7 x is super-sequence of y if y is a subsequence of x 



.4. 



= £ \n'(£-l,m)(fl + im)f(£,m + l,a)x 



m— 1 



log (P + 7m) 



m—0 



1-2 



(J3 + 7 + 7m) x 



(- )™l g(/3 + 7 + 7m) 

1 — a 



m=l v 7 

x log (0.5(1 -a) 

^-2, 



(39) 



with 



(40) 



Bi = a(l - a) log ax 



^[2a 3 (£-2)-(f 2 +£-6)a 2 
1 — a 

+ (£ 2 -3£- 2)a + 2£] , 



£ 2 =a(l-a) k>g(l-a)x 

-1 



(41) 



E K 1 )(^ + 7m)( T ^-r(^-m-2) 
z — ' l\m — 1/ 1 — a 



log(l - a) 
1 — a 



[-2a 3 (£-2) + a 2 (£ 2 +£-6) 



2a(£ 2 -2£ -!)+£(£ -3)] 



(42) 



B 3 = 



m— 1 

= £. 



"£[C~_>+^)<i-»r 



-tn-2 a m+l 



(43) 



Now, we consider the length £+1 output sequences. Obviously, 
for the alternating sequences (i.e., y such that |y| = n r (y)) 
of length £ + 1, we have iV (y) = 0. Denoting by y* the set 
of length £ + 1 non-alternating sequences, for any y G y* the 
duplicated bit can be found in one of the runs of y with a 
length greater than 1. Hence, for any y e J*> we have 

Pv(y)= £ (rj(y)-l)Pi-/(Wy)-l,a) 
r-r j(y)>i 

= (£ + l-n r (y)) Pi -f(l,n r (y)-l,a), 

where the second equality follows from the fact that duplica- 
tion error can not create a new run in the received sequence. 



Thus, we have we have 

n,.(x) 

E PxW E ri(x)logri(x) 
E ^V(y)logPr(y) = g(l-logg)+?(£-l)if 6 (a) xe {o,i}< 



i=i 



y6^* £ I»r(x) 

m— 1 v 7 



m=l x:n r (x)= 



(44) 



= E /(^ a ' m ~ x ) E E ri W logr ^ x ) 

m=l x:n,.(x)=m i=l 

£ f-m+1 

Now, we turn to iJ(Y(X £ )|X £ ). We have = E f(Z,at,m- 1) E n"(k,m,£) -klogk 



k=l 

e-m+i 



H(Y(Xt)\X*) = H b (p,q) + (p + q )logl = £ m/(*,a,m - 1) E ^ J*_ *J ■ klogk 

m=2 k=l V 7 

— 2^ P *( x ) ri(x) log ri(x) . (45) v ; 

xe{o,i)' *=i where the last equality follows from Lemma [3] Putting d36i l. 

<E3, dSJ, (ED, 63. 6D, 63, 63, 6D> 63, 63 together, 

we get L%j. 

Denoting by n"(k, m, £) the number of times a run of length ■ 
k appears in all possible length I sequences containing m runs 



